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PREFACE 


The  intent  of  this  handbook  is  to  orient  newly  assigned 
SAC  aircrew  evaluators  to  the  programs  involved  in  managing 
a  successful  stan/eval  program.  It  is  not  intended  to  make 
anyone  an  expert  in  stan/eval  programs  nor  to  serve  as  a 
sole  source  of  information  for  administering  programs.  Rather 
it  is  an  attempt  to  provide  a  common  sense,  general  overview 
of  the  various  stan/eval  programs  into  which  individuals  can 
plug  specific  facts  extracted  from  the  appropriate  regulations 
Accordingly,  references  where  more  information  can  be  obtained 
for  each  program  are  provided  at  the  end  of  each  chapter. 

This  handbook  is  a  result  of  the  fact  that  information 
governing  stan/eval  programs  is  sometimes  sketchy  and  often 
scattered  among  several  regulations.  Consequently,  where 
specific  guidance  is  available  in  one  reference,  the  subject 
is  treated  lightly  herein  and  the  reader  is  referred  to  the 
specific  guidance.  Where  guidance  is  not  so  specific,  the 
subject  is  treated  in  greater  depth  with  suggestions  and 
points  to  consider  when  administering  the  particular  program. 

Since  an  analysis  of  flight  evaluation  procedures  and 
techniques  is  beyond  the  scope  of  this  handbook,  qualification 
evaluations  are  mentioned  only  to  show  how  they  fit  into  the 
overall  stan/eval  mission  and  how  stan/eval  programs  fit 
into  the  same  mission.  The  emphasis  placed  on  programs  is 
not  intended  to  detract  from  the  importance  of  consistent, 
impartial  flight  evaluations.  Indeed,  the  intent  of  the 
entire  stan/eval  program  would  be  defeated  if  it  were  not 
based  on  objective  data.  However,  doing  justice  to  an  analy¬ 
sis  of  evaluation  procedures  and  techniques  would  require 
another  entire  handbook.  Newly  assigned  evaluators  are 
encouraged  to  read  SACR  50-6,  Chapters  2  and  3,  and  SACR  60-4, 
Vol.  I,  Chapter  10  for  guidance  on  conducting  evaluations. 

This  handbook  is  intended  to  apply  to  all  crew  positions 
in  all  types  of  aircraft  assigned  to  SAC.  If  a  bias  exists 
toward  -135  aircraft  and  the  navigator  crew  position,  it  is 
unintentional  but  due  to  the  author’s  background.  Similarly, 
not  all  information  contained  herein  will  apply  to  Air 
Reserve  Force  (ARF)  units.  Exceptions  for  ARF  units  may  be 
found  in  SACR  60-4,  Vol.  I,  Chapter  15. 


CONTINUED 


Finally,  although  this  handbook  is  not  intended  as  a 
checklist  for  1  CEVG  evaluation  preparation,  with  two 
exceptions,  the  programs  covered  are  arranged  in  the 
groupings  used  by  1  CEVG  for  grading  purposes:  Stan/eval 
organization  and  facilities  are  assumed  to  be  above  the 
responsibility  of  the  newly  assigned  evaluator  and  are 
therefore  omitted.  Stan/eval  office  administration  is 
assumed  to  be  the  responsibility  of  the  branch/division 
chief  and  administrative  personnel  and  is  also  omitted. 
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Chapter  One 


PURPOSE  OF  STAN/EVAL 
OVERVIEW 

"SAC  standardization  and  evaluation  programs  provide 
the  means  to  monitor  and  control  crew  reliability  and  the 
effectiveness  of  the  training  they  receive"  (3:1-1).  This 
dual  purpose  is  accomplished  by  annual  qualification 
evaluations  and  the  programs  described  in  this  handbook. 

QUALIFICATION  EVALUATIONS 

Part  of  the  mission  of  SAC  standardization  and  evalua¬ 
tion  programs  is  to 

evaluate  aircrew  performance  to  determine  crew 
adherence  to  standardized  operating  procedures, 
techniques,  and  capability  to  perform  EWO/ 
contingency/ reconnaissance/ training  assignments 
(6:1-1)  . 

Qualification  evaluations,  as  required  by  and  specified 
in  AFR  60-1,  determine  the  qualification  of  individuals 
to  perform  their  aircrew  duties.  Qualification  evaluations 
are  thoroughly  covered  in  AFR  60-1,  SACR  50-6,  and  SACR 
60-4,  Vol.  I.  This  handbook  examines  the  stan/eval 
programs  that  supplement  qualification  evaluations  in 
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monitoring,  aircrew  reliability  and  evaluate  the  effectiveness 
of  the  training  aircrews  receive. 

STAN/EVAL  PROGRAMS 

"Stan/eval  personnel  must  evaluate  both  the  individual 
and  the  training  he/she  has  received"  (6:16-1).  The 
programs  discussed  in  this  handbook  affect  both  of  these 
functions.  The  no-notice  program  enhances  monitoring 
aircrew  reliability.  Administrative  orograms  facilitate 
both  qualification  and  no-notice  evaluations.  The  trends 
analysis  program  is  the  vehicle  for  detecting  unfavorable 
trends  in  performance  that  reflect  weakness  in  the  training 
program.  The  training  monitoring  program  is  designed  to 
detect  weaknesses  in  the  training  program  before  those 
weaknesses  result  in  aircrew  deficiencies.  The  Training 
Review  Panel  is  where  stan/eval  reports  its  findings  to 
the  commander,  and  where  corrective  action  is  assigned  to 
remedy  training  program  deficiencies. 

The  remaining  chapters  of  this  handbook  are  devoted 
to  examining  the  programs  thar  SAC  has  established  to 
monitor  and  control  crew  reliability  and  the  effectiveness 
of  the  training  crews  receive.  Although  the  programs 
are  examined  individually,  they  all  work  together  and 
cend  to  flow  into  each  other  in  a  smoothly  functioning 
organizat ion . 
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REFERENCES 


The  following  regulations  provide  specific  information 
on  the  purpose  and  responsibilities  of  stan/eval: 

AFR  60-1,  Chapters  1,  4,  5,  6 

AFR  60-1/ SAC  Supp  1 

SACR  50-6,  Chapters  1,  2,  3 

SACR  60-4,  Vol.  I,  Chapters  1,  2,  3 
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Chapter  Two 


THE  NO- NOTICE  PROGRAM 

RATIONALE 

The  program  established  for  conducting  qualification 
evaluations  allows  a  five-month  period  for  crewmembers  to 
complete  all  evaluation  requirements.  After  having  completed 
the  various  prerequisites  and  knowing  in  advance  when  the 
evaluation  will  be  conducted,  the  crewmember  being  evaluated 
may  well  be  better  prepared  to  demonstrate  proficiency  in 
his/her  particular  specialty  than  he/she  would  normally 
be  on  a  day-to-day  basis.  In  order  to  observe  how  crew¬ 
members  perform  routinely,  when  they  haven't  had  advance 
notice  of  the  evaluation,  APR  60-1  provides  for  and 
SACR  60-4,  Vol .  I,  directs  the  establishment  of  no-notice 
evaluation  programs. 


NO -NOT ICE  PROGRAM 

The  different  types  of  no-notice  and  spot  evaluations 
included  in  the  no-notice  program  are  described  in 
SACR  60-4,  Vol.  I,  Chapter  3.  Although  the  types  of 
evaluations  are  spelled  out,  SACR  60-4  does  not  specify 
exact  goals  or  percentages  a  no-notice  program  should 
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accomplish,  nor  does  it  identify  likely  candidates  for  no- 
notice  evaluations.  The  reason  for  this  is  to  allow  units 
the  discretion  to  tailor  the  programs  to  their  specific 
situations.  The  no -not  ice  program  is  intended  as  "a 
management  tool  for  the  unit  commander,”  and  sufficient 
latitude  is  granted  to  make  the  program  as  responsive 
as  possible  to  unit  needs  (6:2-4). 

All  crewmembers  evaluated  in-flight  must  be 
administered  an  emergency  procedures  examination  and  a 
publications  check  within  10  working  days  of  the  flight 
unless  they  have  had  one  within  the  30  calendar  days 
prior  to  the  flight  (6:3-2).  No-notice  flight  evaluations 
require  the  completion  of  an  AF  Form  8.  No-notice  HP 
exams  and  spotchecks  do  not  require  an  AF  Form  8  unless 
corrective  training  is  required  (6:14-1).  No-notice  EP 
exams  that  do  not  require  corrective  training  should  be 
documented  in  the  Remarks  section  of  the  AF  Form  8  that 
reflects  the  individual's  current  qualification  check 
(6  : 14-2)  . 

Program  '.oals 

In  order  to  measure  its  own  effect 'veiv  i  i 
conducting  the  no-notice  program,  stan/eval  should 
establish  program  goals.  Program  goals  serve  as  bench¬ 
marks  to  measure  no-notice  accomplishment  against  and  help 
focus  attention  on  the  program  to  ensure  it  receives 
adequate  emphasis  and  support.  Although  a  common  in-flight 
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no-notice  goal  is  to  evaluate  5(H  of  each  crew  position 
annually,  your  unit  goal  should  reflect  a  level  that 
gives  stan/eval  an  adequate  sampling,  yet  is  realistically 
achievable.  Goals  for  ground  no-notice  and  spot  checks 
vary  widely.  The  same  adequate  but  achievable  guidelines 
apply.  In  all  cases,  any  no-notice  rate  that  provides 
stan/eval  and  the  commander  with  a  valid  indication  of 
day-to-day  crewmember  performance  is  an  effective  goal. 

Whatever  goal  is  selected  should  be  tracked 
separately  for  each  crew  position  so  that  an  adequate 
sampling  is  obtained  for  each  specialty.  Otherwise, 
surpassing  the  goal  for  one  crew  position  could  obscure 
the  fact  that  an  adequate  sampling  was  not  obtained  for 
another  crew  position. 

Another  goal  consideration  is  to  obtain  no-notice 

evaluations  outside  individual  eligibility  periods. 

The  no-notice  evaluation  program  is  not 
intended  to  be  a  substitute  for,  or  a 
replacement  of,  the  scheduled  Qualification/ 

Instrument  evaluation  (6:3-2). 

Conducting  no-notice  evaluations  during  the  time  when 

crewmembers  are  consciously  preparing  for  an  evaluation 

defeats  the  purpose  of  the  program. 

Target  Population 

Closely  tied  to  the  problem  of  establishing  no-notice 
program  goals  is  the  question  of  whom  to  select  for 
evaluation.  The  goal  of  the  program  is  to  sample  the 
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routine  performance  of  a  cross-section  of  crewmembers,  yet 
fairness  and  prudence  demand  that  some  consideration  be 
given  to  evaluate  selection.  Neither  the  program  goal 
nor  fairness  is  served  by  evaluating  the  same  people 
repeatedly.  By  identifying  people  who  have  not  had  a 
no-notice  evaluation  of  similar  activity  or  a  no-notice 
evaluation  of  any  activity  (for  whatever  time  period  is 
appropriate  for  your  situation),  you  can  reduce  feelings 
of  harassment  and,  in  the  long  run,  obtain  a  better  cross- 
section. 

On  the  other  hand,  since  part  of  the  overall  stan/ 
oval  mission  is  to  document  aircrew  qualification, 
there  are  going  to  be  crewmembers  whose  qualification 
you  will  want  to  check  even  though  such  evaluations  might 
reduce  the  randomness  of  your  sampling.  These  evaluations 
are  justified  by  safety  considerations  and  the  commander's 
need  to  be  sure  his  aircrews  can  fulfill  their  missions. 
Examples  of  this  category  include  recent  graduates  of  Combat 
Crew  Training  School  who  have  had  sufficient  time  to  become 
proficient  and  learn  local  procedures  but  are  not  yet 
in  their  qualification  eligibility  periods;  people  who 
have  demonstrated  less  than  satisfactory  performances  on 
previous  evaluations  whose  progress  you  want  to  check; 
and  people  identified  by  squadron  commanders,  flight 
commanders,  and  staff  instructors. 
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REFERENCES 


The  following  regulations  contain  information  on 
the  no-notice  program: 

AFR  60 - 1 ,  Chapter  5 

SACR  60-4,  Vol.  I,  Chapters  2,  3,  14 


9 


Chapter  Three 


ADMINISTRATIVE  PROGRAMS 

RATIONALE 

Although  the  administrative  programs  covered  in  this 
chapter  do  not  contribute  directly  to  evaluating  aircrew 
performance  or  the  quality  of  aircrew  training,  they 
facilitate  both  of  these  goals  and  are  necessary  for  the 
effective  accomplishment  of  stan/eval's  mission.  The 
specific  rationale  for  each  program  and  how  each  program 
supports  the  stan/eval  mission  should  become  apparent 
as  the  programs  are  individually  covered. 

SCHEDULING 

Before  evaluations  can  be  accomplished,  some  type 
of  scheduling  system  is  necessary  to  ensure  that  evaluators 
and  evaluatees  are  available  to  fly  on  a  sortie  that 
includes  required  evaluation  activity  before  the  evaluatee' 
current  qualification  expires.  SACR  60-4,  Vol.  I,  specifies 
a  scheduling  procedure  to  accomplish  this  function.  The 
person(s)  in  stan/eval  responsible  for  scheduling  need 
to  coordinate  with  the  Operations  Systems  Management 
Branch  (DOTF)  and  the  aircrew  scheduling  branches  to 
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get  evaluations  properly  scheduled  and  to  get  DOTF  flight 
records  updated  after  the  evaluation  is  complete. 

Every  week  DOTE  will  provide  stan/eval  with  a 
minimum  of  two  copies  of  the  Flying  Resources  Scheduling 
Action  Report  (6:2-4).  This  report  contains  a  six-month 
listing  of  individual  qualification  expiration  dates. 

The  stan/eval  scheduler  should  use  this  report  to  plan  an 
evaluation  schedule  that  precludes  individuals  from 
exceeding  their  expiration  dates.  No  later  than  the  fifth 
day  of  each  month,  stan/eval  then  gives  the  scheduling 
branches  the  desired  evaluation  schedule  for  the  next 
month  and  completes  preliminary  coordination  of  the 
second  subsequent  month  (7:3-1). 

A  representative  from  stan/eval  must  attend  the 
weekly  pre  60-9  meeting  to  review  and  finalize  the  details 
of  the  next  week's  flying  schedule  (7:3-5).  This  is  the 
last  quality  control  check  before  the  schedule  goes  to 
print,  so  the  proposed  schedule  should  be  checked 
carefully  to  make  sure  all  required  activity  is  included. 
Some  additional  items  to  check  before  and/or  during  this 
meeting  include:  ensuring  the  scheduled  evaluatees  have 
all  their  evaluation  prerequisites  complete;  checking  that 
tactical  crewmembers  are  scheduled  to  be  evaluated  while 
flying  with  their  own  crews;  and  ensuring  that  qualifica¬ 
tion  and  instrument  checks  are  combined  when  appropriate 
(6:3-1  to  3-3). 
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The  stan/eval  scheduler  must  coordinate  with  other 


evaluators  to  stay  informed  of  evaluations  that  are 
incomplete  or  require  rechecks.  Evaluation  activity  that 
was  not  accomplished  needs  to  he  completed  prior  to  the 
evaluatee’s  expiration  date,  and  the  stan/eval  scheduler 
needs  to  coordinate  rescheduling  this  activity  with  the 
scheduling  branch.  Similarly,  rechecks  need  to  be 
accomplished  after  the  corrective  training  is  complete 
but  before  the  corrective  training  due  date  expires. 

The  scheduler  must  coordinate  with  the  agency  conducting 
the  corrective  training  (usually  the  squadron)  and  with 
the  scheduling  branch  to  get  the  recheck  evaluation 
scheduled  within  the  time  constraints  mentioned  earlier. 

After  qualification  evaluations  are  complete,  DOTE 
needs  to  be  advised  so  their  master  records  can  be 
updated.  One  copy  of  the  Flying  Resources  Scheduling 
Action  Report  should  be  annotated  with  all  flight  check 
completion  dates  and  new  expiration  dates.  The  reasons 
for  any  expiration  dates  that  were  exceeded  should  be 
listed  and  the  annotated  text  should  be  returned  to 
DOTF  weekly.  The  following  duty  day,  stan/eval  should 
receive  an  updated  report  from  DOTF  (6:2-4). 

TEST  CONSTRUCTION 

Written  examinations  are  required  for  qualification 
and  must  be  administered  before  the  flight  evaluation  (1 
In  SAC,  unit  stan/eval  organizations  are  responsible  for 
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constructing  the  necessary  examinations  (6:9-1).  SACR  60-4, 
Vol .  I,  Chanter  9,  ant!  SACK  r>0-6,  Chapter  4,  contain 
specific  instructions  on  the  mechanics  of  examination 
construction.  These  regulations  also  clearly  spell  out 
the  content  requirements  for  eme»-"''»u,y  procedures  exams, 
but  1  CEVG  evaluation  results  reports  indicate  some  mis¬ 
understanding  exists  concerning  the  content  of  open  book 
qualification  exams  and  instructor  exams. 

Qualification  hx am s 

Like  any  stan/eval  exam,  open  hook  qualification 
examinations  should  measure  knowledge  applicable  to 
every  crewmember  of  the  particular  specialty  that  is  not 
determined  through  flight  evaluations.  Unlike  emergency 
procedure  exams,  open  book  examinations  do  not  measure 
knowledge  of  critical,  nced-to-know  items.  Rather,  they 
measure  knowledge  of  systems,  malfunction  analysis, 
performance,  procedures,  etc.  In  short,  they  measure  the 
ability  of  crewmembers  to  respond  to  both  normal  and 
abnormal  situations  that  are  not  of  a  critical  nature. 
Consequently,  exam  questions  should  be  pertinent  and 
challenging  enough  to  require  the  average  crewmember 
to  refer  to  the  source  books.  The  purpose  of  the  open 
book  exam  is  not  served  by  asking  questions  that  are 
common  knowledge  or  reflect  situations  that  commonly 
occur  on  any  flight.  On  the  other  hand,  questions 
should  not  be  so  obscure  as  to  be  meaningless. 


Instructor  Exams 


Instructor  examinations  should  challenge  knowledge 
required  to  be  a  successful  instructor.  Questions 
should  pertain  to  instructors  and  instructing,  not 
obscure  facts  that  are  applicable  to  but  less  well- 
known  by  non-  instructors  .  Question  sources  such  as 
SACR  50-24,  SACR  50-46,  the  51-series  regulations,  and 
other  sources  that  apply  to  training  and  instruction  arc 
more  appropriate  than  sources  containing  information  that 
applies  to  all  members  of  the  crew  specialty  regardless 
of  instructor  status.  Instructor  techniques,  required 
training  forms  and  their  completion,  rules  governing 
academic  training,  etc.  are  more  appropriate  areas  for 
instructor  exam  questions  than  systems  questions,  for 
example. 


FLIGHT  EVALUATION  FOLDERS 

Flight  Evaluation  Folders  (FEF)  are  used  to  document 
the  results  of  qualification  and  no-notice  flight 
evaluations,  CPT  checks,  and  any  spot  or  ground  no- 
notice  activity  that  requires  corrective  training.  Since 
the  FEF  is  the  only  complete  record  of  an  individual's 
evaluation  activity,  it  should  be  accurately  maintained 
to  reflect  actual  performances  that  were  observed.  The 
FEF  contains  two  forms,  AF  Forms  8  and  AF  Forms  942. 
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A F  Forms  8 


Although  AFR  60-1  and  SACR  60-4,  Vol.  I,  conti  in 
specific  guidance  for  properly  filling  out  the  AF  Form  8, 
it  is  impossible  to  cover  every  possible  contingency 
in  a  regulation.  For  situations  not  covered  in  the 
regulations,  an  accurate  description  or  the  situation 
that  existed  during  the  evaluation  and  the  performance 
observed  will  provide  a  clear  record  of  the  evaluation. 
The  Additional  Comments  block  of  the  Form  8  can  be  used 
to  clarify  anything  the  evaluator  feels  is  necessary. 

Even  if  a  situation  not  covered  in  the  book  occurs, 
if  a  person  not  already  familiar  with  the  situation 
can  read  the  form  and  determine  what  the  circumstances, 
performance,  and  outcome  were,  the  purpose  of  the  form 
will  have  been  served. 

AF  Form  942 

Since  AF  Forms  8  are  generally  removed  from  the  FEF 
after  two  years,  the  AF  Form  942  is  the  only  permanent 
record  of  evaluation  activity.  Therefore,  it  should 
accurately  reflect  the  individual's  evaluation  history. 
AFR  60-1,  Attachment  1,  and  SACR  60-4,  Vol.  I,  Chapter  14 
contain  specific  instructions  for  filling  out  the  AF  Form 
942  and  the  maintenance  of  Flight  Evaluation  Folders. 
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MANUALS  CONTROL 


As  mentioned  earlier,  stan/eval  is  responsible  for 
evaluating  crewmember  qualification.  Qualification  is  based, 
for  the  most  part,  on  the  correct  performance  of  normal  and 
emergency  procedures.  To  ensure  crewmembers  are  aware  of 
the  most  current  procedures,  they  need  current  flight  manuals. 
Within  SAC,  stan/eval  organizations  administer  the  flight 
manual  program  as  specified  in  AFR  60-9,  SACR  60-4, 

Vol .  I,  and  T.O.  00-5-2  (6:11-1). 

A  wing/base  manuals  control  officer  and  assistant  from 
within  stan/eval  must  be  appointed  and  designated  in  writing. 

The  manuals  control  officer  must  comply  with  T.O.  00-5-1  and 
T.O.  00-5-2  in  establishing  requirements,  and  procuring  and 
distributing  flight  manual  technical  orders  to  flying 
personnel  and  appropriate  wing  staff  agencies  (6:11-1). 

The  manuals  control  program  also  includes  two 
quality  control  functions.  The  manuals  control  officer 
is  responsible  for  ensuring  that  a  current  list  of  flight 
manual  publications  and  operating  procedures  is 
prominently  displayed  within  each  manuals  control  office 
and  flight  planning  area  if  not  located  within  the  same 
facility  (6:11-1).  This  allows  individuals  to  check 
their  personal  flight  manuals  for  currency.  Secondly, 
the  wing/base  manuals  control  officer  is  responsible 
for  inspecting  each  squadron/unit's  manuals  control 
procedures  once  each  month  to  ensure  that  timely  distri¬ 
bution  and  receipting  procedures  arc  being  accomplished  (6:11-1). 
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Specific  instructions  for  managing  the  manuals  control 
program  are  contained  in  SAC R  00-4,  Yol.  I,  Chapter  11. 

Though  not  the  responsibility  of  the  manuals  control 
officer,  stan/cvnl  is  the  unit  point  for  contact  for  the 
submission  of  recommendations  for  changes  to  flight  manuals. 
The  flight  manual  change  program  is  designed  to  get  everyone 
who  uses  flight  manuals  involved  in  keeping  them  as  current 
and  accurate  as  possible.  Anyone  with  a  recommendation  for 
improving,  c.larirying.  or  correcting  information  in  a 
flight  manual  can  and  should  prepare  an  AF  Form  847  and 
submit  it  to  stan/eval.  Someone  in  stan/eval  of  the  same 
crew  specialty  should  evaluate  the  recommendation  ard  dis¬ 
cuss  its  merits  with  the  stan/eval  chief/branch  chief  who 
will  complete  the  first  endorsement.  Three  copies  of  the 
endorsed  form  should  then  be  mailed  directly  to  1  CEVG/ST 
for  further  evaluation  (8:1).  Further  guidance  on  the 
procedures  for  recommending  flight  manual  changes  is 
contained  in  ART-'  60-9  and  SAC  Supp  1  thereto. 

REFERENCES 

The  following  regulations  contain  specific  guidance 
for  the  administrative  programs  listed: 

Scheduling 

SACR  60-4,  Vol .  I,  Chapter  2 

SACR  60-9,  Chapter  3 
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Test  Construction 


APR  60-1 ,  Chapter  5 

SACR  50-6,  Chapter  4 

SACR  60-4,  Vol .  I,  Chapter  9 

Flight  Evaluation  Folders 

AFR  60-1,  Chapter  6,  Attachment 
SACR  60-4,  Vol.  I,  Chapter  14 

Manuals  Control 
AFR  60-9 

APR  60-9/SAC  Supplement  1 
SACR  60-4,  Vol.  I,  Chapter  11 
T. 0.  00-5-1 
T.O.  00-5-2 
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Chapter  Four 


TRENDS  ANALYSIS 

RATIONALE 

For  any  evaluation  administered,  the  corrective 
training  assigned  as  a  result  of  any  discrepancies 
should  bring  that  individual's  performance  back  within 
standards.  "However,  evaluation  of  training  programs  is 
equally  important"  (6:16-1).  If  several  instances  of  the 
same  discrepancy  are  observed,  there  is  a  strong  indication 
of  a  weakness  in  the  training  program  that  needs  to  be 
addressed.  If  such  a  weakness  exists,  improving  or 
amending  the  training  program  can  prevent  other  crew¬ 
members  from  making  the  same  mistake,  resulting  in  an 
overall  improvement  in  aircrew  performance  and  mission 
capability  levels. 


TRENDS 

"A  trend  may  exist  if  two  or  more  individuals  in  a 
relatively  short  time  span  make  the  same  error  while  per¬ 
forming  similar  tasks"  (6:16-1).  Four  operative  words  in 
this  definition  are:  "may,"  "short,"  "same,"  and  "similar." 
These  terms  are  vague  because  it  is  nearly  impossible 
to  quantify  the  definition  of  a  trend;  there  are  too 
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many  variables.  Thus  "may"  indicates  that  the  final  decision 
as  to  whether  a  trend  exists  is  a  judgment  call.  However, 
further  consideration  of  the  operative  words  of  the 
definition  can  help  make  judgment  calls  easier. 

How  short  is  a  "short"  time?  Although  trends  have  to  be 
analyzed  at  least  quarterly,  a  calendar  quarter  is  not  a 
magic  figure  (6:2-5).  Quarters  are  used  as  the  minimum 
for  trend  evaluation  to  ensure  analysis  is  conducted 
routinely  without  increasing  the  frequency  to  the  point  that 
the  data  sample  size  is  insufficient  for  valid  results. 

Some  trends,  however,  can  become  apparent  quickly  within  a 
quarter.  Tf  a  trend  is  noted  prior  to  the  end  of  a  quarter, 
it  should  be  reported  at  the  next  monthly  Training  Review 
Panel  (TRP)  rather  than  waiting  until  the  quarterly  review. 
Trends  can  also  develop  between  quarters.  Errors  observed 
near  the  end  of  a  quarter  may  not  appear  to  be  a  trend 
until  combined  with  similar  errors  in  the  subsequent 
quarter.  Failing  to  consider  discrepancies  from  the  previous 
quarter,  even  though  they  did  not  appear  significant  at 
the  end  of  the  quarter,  can  preclude  identifying  the  trend 
and  correcting  the  training  problem  as  early  as  possible. 
Finally,  the  time  period  cannot  be  divorced  from  the  sample 
size.  Three  similar  errors  in  a  month  may  be  a  trend  if 
they  occurred  during  six  evaluations;  they  may  not  repre¬ 
sent  a  trend  if  they  occurred  during  50  evaluations. 
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Defining  the  "same"  error  seems  fairly  straightforward 
at  first  glance:  two  or  more  people  made  the  same  mistake. 


But,  what  about  the  magnitude  of  the  error? 

To  discriminate  between  a  trend  and  a  number 
of  isolated  incidents,  the  severity  as 
well  as  the  frequency  of  less  than  desirable 
activity  must  be  considered  (6:2-5). 

Thus,  activity  does  not  have  to  be  unqualified  to  be  con¬ 
sidered  in  trends  analysis.  Qualified  with  training  and 
qualified  but  critiqueable  performances  can  also  be  con¬ 
sidered.  Although  inputs  from  these  sources  do  not  reflect 
unqualified  activity,  they  can  help  indicate  possible 
training  program  weaknesses.  How  many  'nstanccs  of  each 
level  or  combination  of  levels  constitute  a  trend  is  still 
a  judgment  call,  but  considering  all  sources  of  data  can 
often  clarify  situations. 

What  are  "similar"  tasks? 

Dat"  for  trend  identification  will  be  derived 
fro.,  any  available  source,  i.e.,  examination 
results,  ground  evaluations  (training  devices, 
instrument  school,  etc.)  and  flight  evaluations 
(6:2-5). 

Although  a  trend  can  be  indicated  by  data  from  these 
various  sources,  the  discrepancies  should  be  similar  enough 
to  define  the  weakness  and  indicate  training  to  correct  it. 
Four  unqualified  navigation  legs  due,  respectively,  to 
DR  errors,  celestial  computation  errors,  celestial  plotting 
errors,  and  use  of  an  unauthorized  aid  probably  do  not 
constitute  a  trend.  However,  three  unqualified  navigation 
legs  due  to  DR  errors  very  well  may  be  a  trend.  On  the 
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other  hand,  three  unqualified  instrument  approaches  flown, 
respectively,  with  TACAN,  rLS,  and  VOR  may  indicate  a  trend 
if  the  problem  is  similar,  for  example,  exceeding  course 
tolerances  due  to  ineffective  crosswind  corrections.  Thus, 
thorough  analysis  is  necessary  to  determine  if  discre¬ 
pancies  arc  similar  enough  to  constitute  a  trend.  This 
does  not  imply  that  high  failure  rates  in  nonrelated  areas 
should  he  ignored.  While  not  a  specific  trend,  such  a 
situation  may  indicate  a  dcficicncv  In  the  overall 
training  program  ((>:!(>- 11. 


ANALYSIS 

A  good  data  collection  and  tabulation  system  can 
simplify  the  process  of  determining  whether  discrepancies 
noted  during  evaluations  fit  t lie  definition  of  a  trend. 

A  good  system  can  prevent  having  to  rely  on  hazy  memories 
or  wading  through  stacks  of  paper  to  discriminate  between 
a  trend  and  a  number  of  isolated  incidents.  Specific 
remarks  about  each  grading  area  should  be  recorded  as  soon 
after  each  evaluation  as  possible.  Remarks  should  be  as 
specific  as  possible  to  allow  comparison  with  other 
discrepancies  to  determine  if  the  same  errors  were  made 
while  performing  similar  tasks.  To  facilitate  comparison, 
some  method  of  cross-referencing  the  remarks  can  save  a 
lot  of  reading  through  extraneous  information.  One  cross- 
referencing  method  that  works  well  is  color  coding 
evaluation  results  by  grading  area  and  date  on  a  summary 
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sheet.  Possible  trends  are  readily  apparent  on  the  summary 
sheet.  Specific  remarks  for  the  evaluation  dates  involved 
can  then  be  analyzed  to  determine  whether  a  trend  actually 
exists.  Some  units  have  computerized  their  trends  analysis 
programs.  Computerized  programs  work  well  when  the  unit 
has  the  evaluation  workload  to  justify  it,  access  to  a 
computer,  and  the  expertise  to  develop  a  computer  program. 
While  a  well-written  computer  program  can  save  considerable 
time,  it  is  not  a  replacement  for  judgment.  Any  system 
that  saves  time  in  accurately  identifying  possible  trends 
can  reduce  stan/eval's  workload,  but,  in  the  end,  the 
determination  of  the  existence  of  a  trend  is  a  judgment 
call . 


REPORTING 

Any  time  a  trend  is  identified,  it  will  be  reported 
to  the  next  TRP  (6:2-5).  SACR  60-4,  Vol.  I,  Chapter  2 
details  how  trends  are  handled  after  they  are  reported  to 
the  panel.  The  trends  analysis  and  correction  process 
can  also  be  aided  by  stan/eval  newsletters  and  notices 
posted  on  squadron  bulletin  boards.  By  advising  crewmembers 
and  squadron  staff  members  of  problem  areas  in  aircrew 
performance,  individual  and  collective  efforts  can  be  made 
earlier  to  reverse  the  trend.  This  does  not  replace  action 
by  the  TRP,  but,  rather,  supplements  it  by  allowing  action 
to  correct  the  problem  as  soon  as  possible.  The  stan/eval 
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chief  and  members  of  the  TRP  must  still  evaluate  any  action 
taken  for  adequacy. 

In  addition  to  identifying  local  unit  trends,  stan/eval 
participates  in  the  SAC  command-wide  trends  program.  To 
accomplish  this,  local  stan/eval  divisions  prepare  and 
submit  the  SAC- DOT (M)  7109  report  to  1  CEVG/ANY  monthly 
(6:16-1).  The  7109  report  compiles  the  results  of  all 
evaluations  administered  during  the  month.  1  CEVG  further 
compiles  these  results  for  the  command  and  analyzes  the 
data  for  command-wide  trends  that  may  not  be  apparent 
at  any  given  unit  or  that  can  best  be  corrected  through 
command  level  training  program  improvements.  Instructions 
for  completing  the  7109  report  arc  contained  in  SACR  60-4, 
Vol.  I,  Chapter  16. 


REFERENCES 

The  following  regulations  provide  guidance  and  informa¬ 
tion  on  the  Trends  Analysis  program: 

AFR  60-1,  Chapter  6 

SACR  50-6,  Chapter  2 

SACR  5 1  - XX ,  Vol.  I,  Chapter  1 

SACR  60-4,  Vol.  I,  Chapters  2,  16 
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Chapter  Five 


TRAINING  MONITORING 

RATIONALE 

One  of  stan/eval's  primary  responsibilities  is  monitoring 
the  effectiveness  of  training  aircrews  receive  (3:1-1).  The 
Trends  Analysis  program  accomplishes  part  of  this  task  by 
monitoring  aircrew  performance  for  adverse  trends  that 
indicate  training  program  deficiencies.  This  process  is 
supplemented  by  the  Training  Monitoring  program.  By  moni¬ 
toring  individual  ground  training  courses  and  training 
device  lessons,  the  quality  of  training  can  be  evaluated  and, 
if  necessary,  improved  before  possible  deficiencies  result 
in  adverse  aircrew  performance. 

ACADEMIC  TRAINING 

Stan/eval  is  responsible  for  evaluating  aircrew 
academic  training  courses  "as  required"  (6:2-5).  Although 
no  evaluation  frequency  or  interval  is  specified,  the 
offices  responsible  for  the  courses  are  required  to 
review  the  lesson  plans  and  course  materials  prior  to  use 
(4:2).  Since  most  course  materials  are  used  during  specific 
scheduling  blocks,  each  course  should  have  been  reviewed 
prior  to  the  schedule  block  in  which  it  is  presented. 
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Thus,  if  stan/cval  monitors  the  course  each  training  block, 
any  changes  made  to  the  lesson  pi  an  and  course  materials 
can  be  evaluated.  Although  reevaluations  of  unsatisfactory 
courses  aren't  specifically  required,  if  the  original  evalua¬ 
tion  is  conducted  early  in  the  scheduling  block,  a  reevalua¬ 
tion  can  be  accomplished  later  in  the  block  to  determine 
if  course  quality  has  improved.  For  training  that  is  not 
accomplished  during  specific  scheduling  blocks,  but  on  a 
random  basis,  periodic  evaluations  provide  a  spot  check 
to  ensure  course  quality  is  adhering  to  standards.  Reeval¬ 
uations  can  be  conducted  as  necessary  when  the  course  is 
presented  again.  For  either  type  of  course  scheduling, 
evaluations  should  cover  any  aspect  of  training  that 
affects  its  quality  such  as  adherence  to  course  objectives, 
instructor  effectiveness,  accuracy,  and  overall  course 
quality. 

I  NSTRUMb.X  T  TRAIN  IN' G 

Because  of  its  potential  impact  on  accident  prevention, 
instrument  training  must  necessarily  he  of  the  highest 
possible  quality.  Therefore,  in  addition  to  evaluating 
academic  training,  stan/eval  must  periodically  inspect 
the  unit  instrument  program  including  the  ground  school 
and  simulator  instrument  training,  if  applicable  (6:2-4). 

The  instrument  ground  school  will  be  inspected  quarterly 
for  compliance  with  the  requ i rements  specified  in  SACR  50-24 
(6:2-4).  Weapon  System  Trainer  instrument  training  will  be 


28 


evaluated  semiannually  for  compliance  with  the  appropriate 
51-series  regulations  and  SACR  50-46  (6:2-5). 

TRAINING  DEVICES 

Stan/eval  is  also  responsible  for  evaluating  "aircrew 
training  devices  and  lesson  plans  in  accordance  with 
SACR  50-46  and  SACR  50-1"  (6:2-5).  The  evaluation  objective 
is  to  determine  the  quality  of  instruction  in  each  type 
of  training  device  and  may  be  accomplished  as  part  of 
the  no-notice  program.  This  is  accomplished  by  evaluating 
a  typical  training  mission  in  each  type  training  device 
assigned  to  the  unit  once  every  other  training  quarter. 
Training  aspects  to  be  evaluated  include  quality  of 
instruction,  lesson  technical  accuracy,  adequacy  of 
lesson  plan  content,  and  training  material  content  (5:3). 
Training  devices  to  be  evaluated  include  WST,  OFT,  CPT, 

IFT,  mission  simulators,  bomb-nav  simulators  (including 
CTD)  ,  mission  trainers  (T-10,  T-4,  T-l),  BOPTT,  and  ARI’TT . 
Egress,  SRAM,  CFT,  and  other  part  task  trainers  need  not 
be  evaluated  by  the  unit  DOV  (5:5).  Areas  of  training 
instruction  which  are  unsatisfactory  during  the  evaluation 
mission  will  be  corrected  and  rechecked  within  seven  days 
(5:4). 


REPORTING 


Written  reports  for  all  evaluations  of  academic 
training  will  be  forwarded  to  the  office  responsible  for 


the  course.  Additionally,  an  information  copy  of  the 
evaluation  report  will  be  forwarded  to  the  DO  or  his 
designated  representative  (4:2). 

Training  device  evaluation  results  will  be  reported 
to  the  unit  DOT,  DON,  or  squadron  commander,  as  applicable, 
depending  on  which  office  is  responsible  for  the  training 
device.  A  sample  format  for  this  letter  is  contained 
in  SACR  50-46,  attachment  8  (5:4). 

In  addition  to  the  above  routine  reporting  procedures, 
any  deficiencies  noted  in  academic,  instrument,  or  training 
device  instruction  must  be  reported  in  writing  directly 
to  the  DO.  Any  deficiencies  noted  must  also  be  discussed 
at  the  Training  Review  Panel  (6:2-5). 

REFERENCES 

The  following  regulations  contain  specific  information 
about  Training  Monitoring: 

SACR  50-24,  Vol .  I 

SACR  50-24,  applicable  volume  to  aircraft  type 

SACR  50-46 

SACR  51-XX,  applicable  volumns  to  type  of  training 

SACR  60-4,  Vol.  I,  Chapter  2 


30 


Chapter  Six 


TRAINING  REVIEW  PANEL 

RATIONALE 

The  unit  commander  is  the  individual  at  the  local 
level  who  is  ultimately  responsible  for  the  training  and 
evaluation  of  aircrews  and  for  insuring  his  aircrews 
are  qualified  to  perform  their  duties  (1:5).  The  Training 
Review  Panel  (TRP)  is  the  forum  where  the  commander  is 
regularly  apprised  of  the  status  of  aircrew  qualification 
and  the  unit  training  program.  Each  wing  office  involved 
with  aircrew  training  participates  in  reviewing  the  wing 
training  program  and  any  actions  that  impact  on  it.  The 
TRP  is  a  totally  integrated  program  for  adjusting  the 
training  program  to  ensure  sufficient  training  of  the  highest 
possible  quality  is  being  accomplished. 

STAN/EVAL  ROLE 

For  stan/eval,  the  TRP  completes  the  loop  between 
evaluation  and  the  unit  training  program.  Stan/eval 
reports  discrepancies  noted  on  evaluations  and  the  panel 
discusses  the  adequacy  of  corrective  action  taken  (6:2-5). 
Additionally,  the  TRP  is  where  trends  are  reported.  The 
panel  reviews  trends  and  the  presiding  officer  assigns 
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corrective  action,  an  OPR,  and  a  suspense  date  to  reverse 
the  trend  (6:2-5).  Possible  corrective  actions  include 
one-time  training  classes,  creating  ongoing  training,  or 
modifying  the  Wing  Directed  Training  Program.  After  the 
corrective  action  has  been  completed,  the  action  agencies 
forward  reports  to  the  stan/eval  chief  who  reviews  them  to 
ensure  the  corrective  training  is  adequate  and  does  not 
conflict  with  existing  directives  (6:2-5).  During 
subsequent  TRP  meetings  the  stan/eval  chief  will  discuss 
the  corrective  action  taken  on  previous  trends  and  the 
panel  will  review  the  status  of  any  previously  identified 
trend  until  it  is  resolved.  Finally,  deficiencies  noted 
in  the  ground  training  or  training  device  programs  should 
be  reported  to  the  panel  for  discussion  so  that  improvements 
can  be  made  and/or  evaluated. 

REFERENCES 

Specific  information  on  the  Training  Review  Panel 
is  contained  in  the  following  regulations: 

SACR  51  - XX ,  Vol .  I,  Chapter  1 
SACR  60-4,  Vol.  I,  Chapter  2 
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